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ABSTRACT 

Communications Link Expert Assistance Resource (CLEAR) is a real time, fault diagnosis 
expert system for the Cosmic Background Explorer (COBE) Mission Operations Room 
(MOR). The CLEAR expert system is an operational prototype which assists the MOR 
operator/analyst by isolating and diagnosing faults in the spacecraft communication link 
with the Tracking and Data Relay Satellite (TORS ) during periods of realtime data 
acquisition . 

This paper discusses the mission domain, user requirements, hardware configuration, 
expert system concept, tool selection, development approach, and system design. 
Development approach and system implementation are emphasized. Also discussed are 
system architecture, tool selection, operation, and future plans. 



INTRODUCTION 

The Communications Link Expert System 
Resource (CLEAR) is an expert system to 
be implemented and demonstrated for the 
Cosmic Background Explorer (COBE) 
Payload Operations Control Center (POCC) 
as a joint project between the Mission 
Operations Division and the Data Systems 
Technology Division within the Mission 
Operations and Data Systems Directorate 
at Goddard Space Flight Center. The 
purpose of this joint project is to design 
and implement an expert system to 
monitor and isolate faults of the COBE and 
Tracking and Data Relay Satellite (TDRS) 
communications link and to provide advice 
to correct these faults. 

BACKGROUND 

NASA's Tracking and Data Relay Satellite 
System (TDRSS) has placed the 
responsibility of configuring, monitoring 
and troubleshooting many types of 
spacecraft communications upon the 
analysts at the consoles in the realtime 
environment of the POCC. The result is a 
complex task which, if not handled quickly 
and properly, can result in poor 
utilization of TDRSS services, inefficient 
spacecraft operations and potential 
hazards to spacecraft health and safety. 

Operating the spacecraft communications 
links with the TDRS requires realtime 
evaluation of a mission oriented subset of 
more than 100 configuration and 
performance parameters and requires 
knowledge of both TDRSS services and 
spacecraft communications systems. This 
evaluation of realtime data must be 
correlated with an understanding of these 
services and systems both to isolate 
problems and to select appropriate 
courses of action to resolve identified 
problems. 

At present, extensive training and 


communication of actual experience are 
used to develop MOR analyst capabilities to 
the fullest. Regardless, the size of the 
task places a large burden on the operator. 
It is the objective of automation, 
specifically utilization of an expert 
system acting as an advisor, to produce a 
more reliable, more efficient and less 
error prone system of operations. 

Spacecraft communication links with the 
Tracking and Data Relay Satellite and 
TDRSS services are used routinely and by 
many missions. This gives the CLEAR 
system a very high utility, particularly if 
only minor modifications are needed to 
allow other missions to use the system. 

The rationale for choosing the COBE 
spacecraft communication links as the 
domain was timeliness. Ground system 
preparation for the mission including 
acquisition of MOR equipment was just 
starting at the time of the decision to 
develop the expert system. This equipment 
included computer workstations of 
sufficient power to support an expert 
system and the designers felt that one 
workstation would normally be available 
during operational periods. 

MISSION DOMAIN 

The COBE, a single observatory mission 
in the area of astrophysics and 
specifically cosmology, will be launched 
in early 1989 and placed in a 900-km 
altitude, circular Sun-synchronous 
terminator (twilight) orbit. The COBE 
will use the TDRSS single access (SSA) 
S-band service for nominal on-orbit 
tracking, command, and telemetry 
support. 

The COBE ground system will acquire data 
via both the Ground Network (GN) and the 
Space Network (SN). The GN will provide 
the interface to the COBE POCC and the 
Sensor Data Processing Facility (SPDF) 
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for the COBE science dump data. The SN 
will provide realtime data acquisition, 
command interface and tracking. 

The Flight Operations Team (FOT), 
operating from the COBE POCC, will 
perform and participate in mission 
planning, realtime telemetry evaluation, 
off-line in-depth analysis, COBE-unique 
data base inputs, and assist in descrepancy 
and enhancement reporting, software 
testing, experiment-scientist interfaces, 
command management and generation, 
orbit and attitude-data coordination and 
data accountability. 

COBE realtime operations will consist of 
four or five 20-minute TDRSS SSA 
forward and return events per day and 
will be used for uplinking stored 
commands, ranging, time management, 
and observatory safety and health 
monitoring. The CLEAR will support these 
periods of realtime operations. 

The COBE POCC will be located in the 
Multi-Satellite Operations Control 
Center (MSOCC). A dedicated COBE 
Mission Operations Room (MOR) will be 
provided. Telemetry data from TDRSS or 
the ground receiving station will enter 
MSOCC via Nascom circuits. The data 
will enter the Telemetry and Command 
(TAC) computer and processed in the 
Application Processor (AP) computer for 
display at keyboard CRT’s in the MOR. 

The COBE Project Flight Operations 
Team performs the function of 
controlling the COBE satellite. MOR 
facility requirements include console 
facilities for three operations positions. 
One of the three positions will be used for 
CLEAR. 

USER REQUIREMENTS 

The following are the functional and 
performance requirements specifications 


for the Communications Link Expert 
Assistance Resource. The information 
herein was extracted by the CLEAR 
knowledge engineers working with 
available TDRSS and COBE 
documentation and with the domain 
expert. 

To carry out its task, the CLEAR 
system will perform the following five 
functions: 

Data Conversion, 

Configuration Checking, 

Communication Link Monitoring, 

Fault Diagnosis, and 
Event Logging. 

Cl FAR is to have no effect upon the COBE 
POCC processing systems and is to be 
transparent to other MSOCC systems. The 
CLEAR system will be a strictly passive 
component of the system supporting COBE 
realtime operations. 

CLEAR is to be transportable within the 
MOR The system will run on any 
Engineering Analysis Workstation (EAW) 
in the COBE POCC without hardware 
modification and with the same operating 
system level software, e.g. 
communication package, graphics routines 
and device drivers, used by other 
application programs. 

CLEAR is to use a standard 
communication package to be developed 
for POCC workstation applications. The 
data furnished by the AP will be ordered 
(positional not keyword) ASCII text 
(alphanumeric not binary). The system 
will extract the TDRSS performance data 
Operations Data Messages (ODM) and 
spacecraft status parameters from the 
communication buffer and convert them to 
the internal format required by the expert 
system. 

COBE and TDRSS configuration parameters 
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for the scheduled event are to be set at 
initialization of the expert system. The 
CLEAR system will allow the operator to 
input values for configuration 
parameters, current date and clock time. 
The system will log the configuration 
parameters for the event. These 
parameters will be utilized by CLEAR to 
check for the start of the event and for 
the correct event configuration and will 
notify the analyst of any discrepancy. 

The analyst is to be able to correct the 
data if the error lies in the CLEAR 
configuration parameters rather than in 
the COBE or TDRSS configuration. After 
the corrected data has been entered, the 
expert system can be reset and restarted. 

CLEAR is to be driven by ODM and status 
data sent by the AP. The system will 
monitor the input data (communication 
buffer arrival) frequency and will warn 
the analyst if input is not received within 
the expected (4 to 5 second) interval. The 
expert system will monitor the 
communications link parameter values to 
detect and isolate faults. 

CLEAR is to diagnose the faults identified 
during an event. The system will 
determine possible sources or causes of a 
fault, rank multiple possibilities in 
order of probability and present the 
result to the analyst. The system will 
also recommend course(s) of action that 
might be taken by the analyst. If 
requested, the system will explain the 
reasoning used to diagnose a fault and 
recommend a course of action. 

CLEAR is to log all expert system activity 
for post event analysis. The system will 
time tag all identified faults and will 
record the inferences, the diagnoses, the 
recommendations offered and the actions 
taken as (and if) indicated by the 
analyst. The system will provide non- 
realtime utilities to print a formatted 


copy of the log, to trace and analyze the 
activity of the expert system during the 
event and to extract statistics for 
evaluation of system performance. 

CLEAR is to operate in realtime with a 
performance requirement derived from 
the expected 3 to 7 second communication 
buffer (input data) arrival frequency. 
The expert system will convert input data, 
check parameter values and perform 
inferences within this time interval. 
Event logging, operator dialog and 
explanations are not realtime events 
subject to the performance requirement. 

HARDWARE CONFIGURATION 

The computer on which the CLEAR will 
run will be any one of the three 
workstations being used for console 
operations in the COBE MOR. These 
workstations are IBM AT compatible 
personal computers that attach to the 
POCC communications network as 
workstations which receive a composite of 
operator display screens. The 
configuration of these AT compatible 
personal computers is as follows: 

• Kandl AT running at 8 mHz clock 
speed, 

• Dolen DC-4 (to be upgraded to DC-8, 
when available) Video Board supporting 
RS-170A, 

• 30 MByte hard disk, 

• 1 .2 MByte and 360 KByte floppy disk 
drives, 

• 1 .5 MByte AST Advantage Memory 
Board, 

• 4 RS-232 communications ports, 

• 1 parallel port, and 

• Color display compatible of 640 x 
480 pixels in 8 colors. 
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This configuration will support the 
Intelligent Systems Corporation (ISC) 
video format for compatibility with 
MSOCC and the POCC display systems. 

The Kandl AT is configured with 512 
KBytes of internal RAM with an additional 
1.5 MBytes of Extended Memory RAM 
available on the AST advantage card. 
Although there is a 640 KByte addressable 
memory limit of PC-DOS 3.1, the 1.5 
MByte AST Advantage Memory Card can be 
configured as a RAM disk and used for 
storing data buffers, or as a repository 
for other executable code that could be 
swapped into the DOS memory area on 
demand. 

The Kandl AT will contain a 30 MByte hard 
(fixed) disk for storage use. The 
operating system and other support 
programs are predicted to occupy 
approximately 10-15 MBytes of disk 
storage. This leaves ample storage for the 
entire CLEAR application and supporting 
files which are predicted to be less than 
one MByte of disk storage. The amount of 
memory required by the CLEAR Event Log 
is not included in this figure as it is 
presently not known how much storage 
will be required. 

EXPERT SYSTEM CONCEPT 

The CLEAR expert system is to assist the 
analyst in the MOR in operating the COBE 
spacecraft communication links with the 
TDRS. The following functions are to be 
performed by the system: 

• monitor the spacecraft 
communication data, 

• isolate suspected faults for analysis, 

• diagnose actual faults, 

• determine the set of alternative 
actions, 


• rank and display the possible 
responses for the analyst, 

• explain the diagnosis, the selection of 
alternatives and the ranking of possible 
responses, and 

• activate the operator selected 
response (future enhancement not part 
of initial prototype). 

Two very significant requirements are the 
"realtime" response required of the 
expert system and the mandatory 
requirement that the effect of the CLEAR 
on the operational system be either nil 
or minimal. These two requirements 
considered together have generated the 
concept of the CLEAR expert system 
prototype attached to the operational 
system as if it were an operator's 
workstation display. This approach, 
rather than that an embedded or inline 
system, is expected to reduce the effect of 
the prototype expert system on the 
operational system to the minimum and to 
meet the response requirement. 

TOOL SELECTION 

The realtime response required of the 
CLEAR system translates into a 
performance requirement for the expert 
system. The data driven and diagnostic 
nature of the expert system place 
interface and inference logic requirements 
on the tool selected to build the 
application. Further selection criteria 
come from the hardware and software 
compatibility requirements. The CLEAR 
must run on the POCC workstation which 
is an IBM AT compatible personal 
computer and must use operating system 
level software written in C and compiled 
under Microsoft® C Version 4.0. 

A number of secondary (desireable rather 
than mandatory) requirements are also 
used in the selection including cost, 
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number of tool users, length of tool usage, 
stability of supplier, development 
environment and availability of source 
code. The secondary selection criteria are 
used to rank the expert system building 
tools that satisfy the mandatory 
requirements. 

Although several commercially available 
expert system building tools meet the 
mandatory requirements based upon 
available information including 
independent benchmark tests, product 
reviews and reported user experience in 
using these products, none is ranked 
higher than a NASA expert system building 
tool, CLIPS. 

CLIPS, a tool for the development of 
expert systems, was created by the 
Artificial Intelligence Section of the 
Mission Planning and Analysis Division at 
NASA/Johnson Space Center. CLIPS 
provides an inference engine and language 
syntax which provides the framework for 
the construction of rule-based systems. 

CLIPS was entirely developed in C for 
performance and portability. The key 
features of CLIPS are: 

•Forward Chaining BuifiS 

• Powerful Rule Syntav; CLIPS allows 
free form patterns, single and multi- 
field variable bindings across patterns, 
user defined predicate functions on the 
LHS of a rule, and other powerful 
features. 

• Portability: CLIPS has been installed 
on over half a dozen machines with 
little or no code changes. 

• Hioh Performanrp- CLIPS' perfor- 
mance on minicomputers (VAX, SUN) is 
comparable to the performance of high 
powered expert system tools in those 
environments. On microcomputers, 
CLIPS outperforms most other 
microcomputer based tools. 


• Embeddable: CLIPS systems may be 
embedded within other C programs and 
called as a subroutine. 

• Interactive Development: CLIPS 
provides an interactive, text oriented 
development environment, including 
debugging aids. 

• Completely In tegrated With C- Users 
may define and call their own functions 
from within CLIPS. 

• Extensible: CLIPS may be easily 
extended to add new capabilities. 

• Source Code: CLIPS comes with all 
source code and can be modified or 
tailored to meet a specific users’ needs. 

• Fully Documented- CLIPS comes with 
a full reference manual complete with 
numerous examples of CLIPS syntax. 
Examples are also given on how to 
create user defined functions and CLIPS 
extension. A User's Guide to introduce 
expert system programming with 
CLIPS is also available. 

CLIPS is available through: 

Computer Software Management and 
Information Center (COSMIC) 

NASA Software Dissemination Center 
University of Georgia 
Athens, Georgia. 


DEVELOPMENT APPROACH 

The CLEAR system is being implemented in 
three phases. The first phase, which was 
completed February 28 , 1987 , was the 
rapid prototyping phase. The prototype 
was developed on a Symbolios 3640 Lisp 
Machine using the Automated Reasoning 
Tool (ART). The first phase demonstrated 
the expert system technology and an 
understanding of the problem domain 
using an advanced development 
environment. The products of this phase 
that transfer to the next are the user 
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interface and the knowledge base. 

The second phase of implementation is the 
operational prototyping phase. This 
operational prototype is being developed 
on the Kandl AT using CLIPS and the 
Microsoft® C 4.0 compiler. In this phase, 
the CLEAR team will evaluate hardware 
and software performance, and locate 
problems using the actual operational 
environment driven by simulated 
operational data. In addition to the early 
determination and resolution of problems, 
the knowledge base is being enhanced and 
the entire system will transfer to the final 
phase. 

The third and final phase of the CLEAR 
system implementation is the installation 
in the COBE MOR of the hardware and 
software developed in the second phase. 
Deferred components and enhancements 
stemming from the second phase will be 
developed at this time. This operational 
prototype will be integrated, tested and 
available to assist the MOR operator/ 
analyst during COBE operations. 

During the first two phases, CLEAR 
receives simulated data from a locally 
written Data Simulator. The simulator 
transmits data resembling the data that 
CLEAR will receive from the MSOCC 
Applications Processor after installation 
and integration in the third phase. The 


CLEAR team developed the Data Simulator 
to allow testing and debugging of the 
expert system without having to wait for 
simulated test data in the last phase. 

The Data Simulator is a software program 
that resides on a VAX 8600. This design 
prevents the simulator from interfering 
with the processing time of the computer 
on which CLEAR is running. The design 
also allows the simulator to be used to test 
CLEAR on both the Symbolics 3640 in the 
first phase and the Kandl AT in the second 
without rewriting or transporting the 
software program. 

In the third phase, CLEAR will have a 
physical interface with the MSOCC 
Applications Processor (AP). 

SYSTEM ARCHITECTURE 

The CLEAR system software architecture 
consists of the Expert System and two 
Interface Subsystems (Figure 1). 

The Expert System is a forward- 
chaining, rule-based system. It is 
implemented using the C Language 
Integrated Production System (CLIPS), an 
expert system development tool developed 
by Johnson Space Center. CLIPS was 
chosen because it is forward-chaining, 
portable, and supports integration of 
external functions written in the 
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CLEAR Software Architecture 
Figure 1 
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programming language 'C' . 

The User Interface subsystem interfaces 
the user’s CRT display to the Expert 
System. This subsystem generates all of 
the text and graphics to be displayed to the 
user on the CRT. Independently developed 
Video Interface Routines are utilized by 
this subsystem to produce the screen 
display. 

The Data Interface subsystem interfaces 
the Expert System to the Applications 
Processor in the Multi-Satellite 
Operations Control Center (MSOCC). This 
subsystem buffers the ODM and spacecraft 
status parameters received from the AP 
utilizing the workstation's 
communications interface software which 
is being developed independently. The 


buffered data is then converted to the 
format required by the Expert System and 
passed to it. 

These three primary subsystems of the 
CLEAR system can be further broken down 
into their functional modules (Figure 2). 

Expert System 

The Expert System consists of fact bases, 
a rule bases, an Inference Engine and an 
Event Log. 

Data enters the Expert System as CLIPS 
facts asserted by the Data-Conversion 
routine. Each fact represents a piece of 
information which has been asserted into 
the Fact Base or Deduced-Data base. In 
CLIPS, the Fact Base is properly called a 
"Fact- List" and the existence or non- 
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existence of facts in this list causes rules 
in the Rule-Base to fire. The actions of 
rules can cause facts to be asserted in or 
retracted from both the Deduced-Data base 
and the Fact Base. 

The Event Log is a log of all Expert 
System activity for post-event analysis. 
The system time tags all identified faults 
and records the inferences, diagnoses, and 
recommendations offered to the analyst. 
The system provides non-realtime 
utilities to print a formatted copy of the 
log, to trace and analyze the activity of the 
inference during the event, and to extract 
statistics for evaluation of system 
performance. 

The Deduced-Data base is a list of facts 
that are deduced by the Expert System, 
such as the status of the links and control 
information. These facts cause rules in 
the User-Interface Rule Base to fire 
which, in turn, sends function calls to the 
User Interface subsystem. 

User Interface 

The User Interface Functions manage the 
screen display. These functions utilize the 
Video Interface Routines that are being 
independently developed to drive the Dolen 
DC-4 (and, subsequently, the DC-8) 
video board. 

Data Interface 

The ODM and spacecraft status parameters 
coming from the AP will be formatted by 
the Terminal Emulator package for 
viewing upon the workstations' CRT by the 
FOA. The Terminal Emulator package 
functionally processes the incoming data 
as follows. 

The parameters enter the POCC 
workstation through the communications 
port and are stored in a circular character 
buffer. Every 30 milliseconds (30 |x- 


sec), a process checks this buffer for 
newly arrived data. When a parameter 
arrives, this process first decodes its 
value, attributes and screen coordinates, 
and then stores them in a video buffer 
named Display Page 0 (zero). 

The Incoming Parameter Monitor will poll 
the Display Page 0 for the new parameters 
and will send them to Data-Conversion 
when located. Data-Conversion converts 
these parameters to the corresponding 
CLIPS facts and asserts them into the 
Expert System. 

A second source of input data is the 
Initialization Table. This table contains 
COBE spacecraft and TDRS configuration 
parameter values. When first set or when 
modified, these values are sent to Data- 
Conversion to be converted to the 
appropriate CLIPS format. If parameters 
in the Initialization Table are modified 
during an event, CLEAR can be reset and 
restarted using these modified values. 

The Expert System, written in CLIPS, 
was the first subsystem to be coded. The 
coding process was straightforward due to 
the similarities between CLIPS and ART 
(in which the phase one prototype was 
written). The User Interface was the 
second subsystem to be developed. A 
highly functional user interface assisted 
in developing knowledge base because it 
was only through the CRT display that the 
capabilities of the expert system could be 
demonstrated to the "expert” for approval 
or refinement. The Data Interface was the 
final subsystem to be coded. Both the Data 
Interface and the User Interface 
subsystems were coded in Microsoft® C 
4.0. 

OPERATION 

As a data-driven expert system, CLEAR 
receives ODM and status data from the AP 
in realtime, monitors the data, advises 
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the FOT of problems and recommends 
possible solutions for each problem 
isolated. To facilitate greater acceptance 
and ease of use, user input is minimal. 

The only user input required by CLEAR is 
the pre-event initialization of parameters 
in the Initialization Table. Default 
settings for each spacecraft and ground 
system parameter are provided along with 
an override via keyboard entry. 

During a pass, the parameter values of the 
Initialization Table can be modified by the 
Flight Operations Analyst (FOA) if the 
expert system detects and notifies him/ 
her of anomalies between these 
parameters and the actual COBE or TDRS 
configuration. The FOA may then reset and 
restart the expert system using the 
corrected parameters. 


If, on the other hand, the user notices an 
incorrect value in the Initialization Table 
before the expert system isolates it, a 
correction can be made without affecting 
the Data Interface software or the 
functioning of the expert system. 

CLEAR outputs to the CRT of the 
workstation. Figure 3 shows the screen 
display of the user interface of the CLEAR 
first phase prototype which was developed 
on the Symbolics 3640. Due to the 
differences in graphics capabilities 
between the Symbolics and the Kandl AT, 
the actual screen display developed in the 
second phase may be different; however, 
the CLEAR team will strive to develop a 
screen display as similar as the graphic 
routines will efficiently permit. 

The display has a COBE-POCC network 
graphic, "Problem" and "Advice" message 
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areas, and various graphic indicators 
which are used for continual display of the 
time and other important parameters. 
The "Problem" and "Advice" message areas 
remain blank until the need for a message 
is determined by the Expert System, at 
which time the problem and the 
appropriate advice are displayed. The 
"Problem" area displays all the problems 
isolated, prioritized by the most likely 
initial cause. The "Advice" area provides 
recommendations on how to correct the 
most critical problem of the "Problem" 
area. In the case of multiple advice 
options, the first one listed is the best 
option, followed by other advice in 
descending order of probable 
effectiveness. 

The COBE-POCC network graphic shown in 
Figure 3 consists of two TDRS spacecraft, 
the COBE spacecraft, and the NGT/WSGT 
and MSOCC boxes. When solid lock on 
COBE is achieved, there are two green 
lines (each line denotes both the forward 
and return links); one from COBE to the 
appropriate TDRS, and one from TDRS to 
WSGT. If data indicates that there is a 
problem with either of these links, the 
troubled link will turn red and flash 
while the other healthy link remains 
green. 

FUTURE PLANS 

The second phase, implementation of an 
operational prototype on the EAW, will be 
finished in the Fall of 1987. The third 
phase will begin at that time and will 


include the following tasks: 

• implementation of deferred modules, 

• addition of enhancements identified in 

the second phase, 

• generation of system documentation, 

• generation of training manuals, 

• delivery and installation of the CLEAR 

prototype in the MOR, 

• system integration, and 

• system test and acceptance by the 

user. 

One additional task, the system 
performance evaluation, must be deferred 
until COBE is launched and a baseline of 
operational experience has been obtained. 
This task is to evaluate both the 
efficiency of the system and the 
effectiveness of CLEAR in assisting the 
operator in the MOR. 
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